Multimodal artificial intelligence (AI) systems have the potential to enhance clinical decision-making by interpreting various types of medical data. However, the effectiveness of these models across all medical fields is uncertain. Each discipline presents unique challenges that need to be addressed for optimal performance. This complexity is further increased when attempting to integrate different fields into a single model. Here, we introduce an alternative approach to multimodal medical AI that utilizes the generalist capabilities of a large language model (LLM) as a central reasoning engine. This engine autonomously coordinates and deploys a set of specialized medical AI tools. These tools include text, radiology and histopathology image interpretation, genomic data processing, web searches, and document retrieval from medical guidelines. We validate our system across a series of clinical oncology scenarios that closely resemble typical patient care workflows. We show that the system has a high capability in employing appropriate tools (97%), drawing correct conclusions (93.6%), and providing complete (94%), and helpful (89.2%) recommendations for individual patient cases while consistently referencing relevant literature (82.5%) upon instruction. This work provides evidence that LLMs can effectively plan and execute domain-specific models to retrieve or synthesize new information when used as autonomous agents. This enables them to function as specialist, patient-tailored clinical assistants. It also simplifies regulatory compliance by allowing each component tool to be individually validated and approved. We believe, that our work can serve as a proof-of-concept for more advanced LLM-agents in the medical domain.

Background: Immune checkpoint inhibitors (ICIs) have revolutionized cancer treatment but can result in severe immune-related adverse events (IrAEs). Monitoring IrAEs on a large scale is essential for personalized risk profiling and assisting in treatment decisions. Methods: In this study, we conducted an analysis of clinical notes from patients who received ICIs at the Tel Aviv Sourasky Medical Center. By employing a Natural Language Processing algorithmic pipeline, we systematically identified seven common or severe IrAEs. We examined the utilization of corticosteroids, treatment discontinuation rates following IrAEs, and constructed survival curves to visualize the occurrence of adverse events during treatment. Results: Our analysis encompassed 108,280 clinical notes associated with 1,635 patients who had undergone ICI therapy. The detected incidence of IrAEs was consistent with previous reports, exhibiting substantial variation across different ICIs. Treatment with corticosteroids varied depending on the specific IrAE, ranging from 17.3% for thyroiditis to 57.4% for myocarditis. Our algorithm demonstrated high accuracy in identifying IrAEs, as indicated by an area under the curve (AUC) of 0.89 for each suspected note and F1 scores of 0.87 or higher for five out of the seven IrAEs examined at the patient level. Conclusions: This study presents a novel, large-scale monitoring approach utilizing deep neural networks for IrAEs. Our method provides accurate results, enhancing understanding of detrimental consequences experienced by ICI-treated patients. Moreover, it holds potential for monitoring other medications, enabling comprehensive post-marketing surveillance to identify susceptible populations and establish personalized drug safety profiles.

Breast cancer is a significant public health concern and is the leading cause of cancer-related deaths among women. Despite advances in breast cancer treatments, medication non-adherence remains a major problem. As electronic health records do not typically capture patient-reported outcomes that may reveal information about medication-related experiences, social media presents an attractive resource for enhancing our understanding of the patients' treatment experiences. In this paper, we developed natural language processing (NLP) based methodologies to study information posted by an automatically curated breast cancer cohort from social media. We employed a transformer-based classifier to identify breast cancer patients/survivors on X (Twitter) based on their self-reported information, and we collected longitudinal data from their profiles. We then designed a multi-layer rule-based model to develop a breast cancer therapy-associated side effect lexicon and detect patterns of medication usage and associated side effects among breast cancer patients. 1,454,637 posts were available from 583,962 unique users, of which 62,042 were detected as breast cancer members using our transformer-based model. 198 cohort members mentioned breast cancer medications with tamoxifen as the most common. Our side effect lexicon identified well-known side effects of hormone and chemotherapy. Furthermore, it discovered a subject feeling towards cancer and medications, which may suggest a pre-clinical phase of side effects or emotional distress. This analysis highlighted not only the utility of NLP techniques in unstructured social media data to identify self-reported breast cancer posts, medication usage patterns, and treatment side effects but also the richness of social data on such clinical questions.

Most melanoma deaths happen to individuals originally diagnosed with early-stage melanoma and then encountered a recurrence that is often not recognized until it has spread or metastasized. Researchers at Massachusetts General Hospital (MGH) recently devised an artificial intelligence-based method for determining which patients are most likely to have a recurrence and benefit from aggressive treatment. The study was published in the journal npj Precision Oncology. Most individuals with early-stage melanoma are treated with surgery to eliminate cancerous cells. Still, patients with more advanced cancer are frequently treated with immune checkpoint inhibitors, which efficaciously enhance the immune response against tumor cells but have serious side effects.

The forecasting tool assesses multiple patient-specific biological and clinical factors to predict the degree of response to immune checkpoint inhibitors and survival outcomes. It markedly outperforms individual biomarkers or other combinations of variables developed so far, according to findings published in Nature Biotechnology. With further validation, the tool may help oncologists better identify patients most likely to benefit from ICB. Discerning, prior to treatment, patients for whom ICB would be ineffective could reduce unnecessary expense and exposure to potential side effects. It could also indicate the need to pursue alternate treatment strategies, such as combination therapies. "It's important to know which treatment modalities patients are most suited for," said Dr. Chan, director of Cleveland Clinic's Center for Immunotherapy & Precision Immuno-Oncology.

Experts trained a computer to tell which patients with skin cancer may benefit from drugs that keep tumors from shutting down the immune system's attack on them, a new study finds. Led by researchers from NYU Grossman School of Medicine and Perlmutter Cancer Center, the study showed that an artificial intelligence (AI) tool can predict which patients with a specific type of skin cancer would respond well to such immunotherapies in four out of five cases. Specifically, the study examined patients with metastatic melanoma, skin cancer that has the capacity to spread to other organs and kills 6,800 Americans each year. The results are important, say the study investigators, because while the drug class studied, immune checkpoint inhibitors, has been more effective for many patients than traditional chemotherapies, half of patients do not respond to them. Adding to the urgency of efforts to determine which patients will respond, researchers say the drugs may cause side effects in many of them and are also expensive.

In recent years, immunotherapy has dramatically improved the treatment options in various cancers increasing the survival rates for treated patients. Among the most promising immunotherapeutic approaches there is the pharmacological manipulation of the physiologic immune checkpoints [1,2,3,4]. Immune-checkpoint blockade is the basis for the clinical antitumor activity of the most promising currently approved antibodies targeting the checkpoint molecules CTLA4 (Cytotoxic T-Lymphocyte Antigen 4), PD1 (Programmed Cell Death 1) and PD-L1 (Programmed cell death ligand 1).Nevertheless, there are heterogeneous response rates to immune checkpoint inhibitors (ICI) [4,5,6] among the different cancer types, and also in the context of patients affected by a specific cancer.